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The New Instructional Leadership: 

Creating Data-Driven Instructional Systems in Schools^ 

Richard Halverson, Jeffrey Grigg, Reid Prichett, and Chris Thomas 



In June 2005, the New York City Public Schools announced that fifth-grade test scores 
had made impressive gains across the city schools — 15.2 percentage points (in students testing 
proficient and above) in math and nearly 20 percentage points in reading. Some of the most 
impoverished, lowest achieving schools were responsible for the largest gains. While politicians 
and policy makers wrangled to claim credit or to question the legitimacy of the results, school 
leaders, teachers, parents, and students offered a simpler explanation: “hard work.” But what did 
they mean by hard work? Leaders and teachers emphasized “a relentless focus on literacy and 
math” and a “ceaseless scrutinizing of tests, quizzes and writing samples” to understand what 
students didn’t know (Herszenhorn & Saulny, 2005). Others highlighted after-school tutoring 
and preparation, improved attendance, pre-kindergarten, smaller classes, fear of grade retention, 
community outreach, and intense political pressure to succeed. However, neither leaders, 
teachers, nor parents could “agree on any one primary reason for the gains.” In part, this was 
because there was no “one primary reason,” no silver bullet, to explain the gains. The gains did 
not result from a particular program, a new policy, or new personnel. Rather, the broad 
improvements in student learning achieved in New York and in schools across the country were 
the results of many factors designed to work together in complex instructional systems. 

The capacity of school leaders and teachers to reshape moribund schools into 
organizations able to respond to the feedback of standardized testing represents a significant step 
forward in our understanding of school leadership practice. The purpose of this paper is to 
provide a framework — the data-driven instructional system (DDIS ) — to help explain how school 
leaders develop this new organizational capacity and to discuss how well the framework captures 
the practices of innovative school leaders, like those in New York. 

The work of school leadership is undergoing a revolution. The recent policy press for 
standards and accountability has led policy makers and the public to hold teachers and schools 
responsible for improvements in student learning. While teachers are ultimately responsible for 
improving student learning in schools, changing the organizational conditions for improvement 
across schools is the central task of school leaders. The new instructional leadership pushes 
beyond the debates between instructional, managerial, and transformational practice to a new 
conception of creating accountable learning systems in schools. And in many schools throughout 
the country, evidence is mounting that leaders are helping schools change practices to improve 
student learning as reflected by standardized testing. Here, we address two questions: What are 
successful school leaders doing to systematically improve student learning? And how can we 
communicate these innovative practices to leaders looking for ways to improve learning? 



* This paper was prepared for the July 2005 annual meeting of the National Council of Professors of Educational 
Administration in Washington, D.C. The authors are grateful for the capable and astute assistance of Jeffrey Watson, 
Cathy Loeb, and Erica Halverson. Address all correspondence regarding this paper to: Richard Halverson, 

University of Wisconsin-Madison, Educational Leadership and Policy Analysis, 1025 W. Johnson St., Rm. 1 186G, 
Madison, WI 53705. 



3 




The New Instructional Leadership 



The initial stage in this change has been widely characterized by researchers and 
educators alike as a move toward data-driven decision making spurred by No Child Left Behind 
(NCLB; 2002). Recent discussions of data-driven decision making have focused on the 
generation, analysis, and use of student achievement data as the primary response to 
accountability demands (Bernhardt, 1998; Holcomb, 1999; Johnson, 2002; Love, 2002). State 
education leaders rush to provide the disaggregated student data from achievement tests 
calibrated to content standards. Private vendors make millions on the sale of data warehousing 
and reporting systems to sort through the data (Burch, 2004). Local districts and service agencies 
hire consultants for data analysis workshops and data retreats to make sense of achievement data 
in the context of local priorities. Schools shop among comprehensive school reform plans and 
packaged curricula already coordinated with state learning standards and assessments. The 
seemingly very recent debate about the legitimacy of standardized testing as a measure of student 
learning has quickly given way to an onslaught of interest in helping schools meet the challenge 
of improving student test scores across systems. Together, tests, standards, and policies make up 
accountability systems. Hanushek and Raymond (2002) argue that these accountability systems 
will help reshape local practices: 

[A] focus on student outcomes will lead to behavioral changes by students, teachers, and schools 
to align with the performance goals of the system. Part of this is presumed to be more or less 
automatic (i.e., public reporting of outcomes will bring everybody onto course with those 
outcomes). But part also comes from the development of explicit incentives that will lead to 
innovation, efficiency and fixes to any observed performance problems, (p. 81 ) 

The inability of schools to adapt easily to the demands of accountability systems suggests 
that such systems exist at multiple levels. From the perspective of schools, these policy 
initiatives constitute external accountability systems. The central problem for guiding systemic 
change is that schools already have robust locally driven accountability systems. Implementing 
the policies of the external system in schools is not merely a matter of providing the proper 
incentives. All schools already engage in many forms of data-driven decision making that rely on 
quantitative data on student attendance, grading, budgets and student placement and on 
qualitative data on teacher, student, and parent reputations. Taken together, these data comprise 
robust internal accountability systems that already shape and guide complex systems of practices 
in schools. Internal accountability systems enable schools to respond to emergent challenges 
while maintaining stable, persistent structures. These internal accountability systems have 
evolved along with the traditional practices of schools, and they help form school cultures that 
provide seamless feedback on core instructional and managerial practices. The press for data- 
driven decision making, then, is not a call for schools to begin to use data, but a challenge for 
leaders to reshape the central practices and cultures of their schools to react intentionally to the 
new kinds of data provided by external accountability systems. 

Thus, the heart of the new instructional leadership is the ability of leaders to shift schools 
from a culture of internal accountability to meet the demands of external accountability. Data- 
driven decision making can produce its intended effects only if supported by organizational 
capacity that allows school teachers and leaders to intentionally change instructional practices in 
the face of new information. As described by Wilson (2004), coherent accountability systems 
depend on establishing a two-way information flow connecting classroom practice and external 
accountability measures. To realize this goal, leaders need to reframe the traditional sense- 
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making practices of schools to respond to a culture of external accountability. This new work, to 
be sure, draws on traditional practices of program and teacher evaluation, curriculum design, 
professional development, and creation of cultures of learning. However, these old tools and 
practices need to be used to challenge the status quo organization of traditional schooling. 
Leaders working in schools traditionally characterized by loose coupling of administrative and 
teaching practice, teacher autonomy, individualized professional development, and unmonitored 
instructional quality now face policy expectations that push for tightened coupling of teaching 
and leadership, teacher collaboration, professional learning aligned with instructional goals, and 
closely monitored instructional outcomes. The new instructional leadership will require 
knowledge and frameworks to guide leaders in creating schools that systematically improve 
student learning. Leaders will need to be able (a) to work with teachers to help students test well, 
while not reducing learning to testing, and (b) to justify changes in instructional and personnel 
practices to an increasingly well-informed community. 

Data-Driven Instructional Systems (DDIS) 

In this paper, we present a theory for understanding the new instructional leadership. We 
realize that, as usual in education research, practitioners are not waiting for researchers to 
provide tools for solving the problems of practice. And of course, some of the new solutions 
“game” the learning process by emphasizing the requirements of testing over authentic learning. 
But that is only part of the picture. It is important not to miss the story of leaders who are 
reinventing instructional leadership and reframing what learning means in our schools. Our 
research addresses the nature of these new practices with a blend of deductive and inductive 
process. We have developed a data-driven instructional systems model to describe the structures 
and practices leaders use to develop their schools’ capacity to use data for instruction. Building 
on research in distributed leadership and organizational learning theory, the DDIS framework 
describes how leaders create systems to re-culture schools for accountability. A successful DDIS 
helps to translate summative achievement test data into formative data teachers and students can 
use to improve teaching and learning. The DDIS framework we describe here was developed 
from a review of the literature on data-driven decision making and refined in the analysis of our 
fieldwork. We begin by presenting the six functions of DDIS and discussing how these functions 
operate together to translate summative data into formative data. We then present a preliminary 
analysis of data we have collected from four schools over the past year to illustrate how the 
DDIS framework functions in schools. We conclude with a discussion of the relative merits of 
shifting instructional leadership from traditional functions to the creation of DDIS in schools. 

Distributed Leadership 

Distributed leadership theory has recently emerged as an analytical tool for understanding 
leadership practice in complex settings (Spillane, Halverson, & Diamond, 2004; Bennett, Wise, 
Woods, & Harvey, 2003). The theory is a descriptive tool that considers how leadership tasks are 
distributed socially and situationally in schools. The central task of instructional leadership is to 
create the conditions for improving teaching and learning in schools. While social distribution 
addresses how leaders delegate and share tasks across actors, situational distribution considers 
how leaders structure contexts to support intended practices (Halverson, 2003). Situational and 
social distribution build on each other: social task distribution helps to create the structural 
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supports that, in turn, establish the traditions and the culture of practice in schools. The concept 
of artifact plays a key role in understanding how professional community is developed in schools 
(cf. Norman, 1993; Simon, 1996). From a school leadership perspective, artifacts refer to 
designed programs, procedures, and policies intended to shape or reform existing organizational 
practices (Halverson, 2001; Halverson & Zoltners, 2001). Taken together, the network of 
artifacts in a given school establishes a system of practice (Halverson, 2003). Tracing the 
development and interaction of artifacts in a local system of practice provides a path for 
researchers and practitioners to access the elusive context of practice for teaching and learning. 

Data-driven instructional systems are systems of practice dedicated to making summative 
data on student learning useful for improving teaching and learning in schools. Considering data- 
driven decision making from a distributed leadership approach requires that we first identify the 
key tasks involved in a DDIS, then identify the structures leaders develop and use to share tasks 
among people and across artifacts. Following DDIS development provides access to the obscure 
and ill-defined process of capacity development for data use in schools. The resultant stories of 
how leaders and teachers develop organizational capacity open a window into how leaders frame 
and solve the complex problems of the new instructional leadership. 

Organizational Learning Theory 

A DDIS helps translate the results of summative testing into formative information 
teachers can use to improve instruction. The system needs to access data, help teachers and 
leaders reflect on what the data mean, plan and determine what needs to be done, and assess the 
results of the plan. In many ways, a DDIS reflects the central concepts of the organizational 
quality movement inspired by W. Edwards Deming. Deming’s ideas inspired organizations to 
move beyond bottom-line results to focus on embedding quality cycles throughout the 
organization (cf. Walton, 1986; Deming, 2000). The quality approach has been simplified by 
embedding redundant plan-do-study-act cycles within organizational processes to ensure that 
feedback on core practices is integrated into the system. Programs such as the Baldridge National 
Quality Program ( http://www.baldrige.nist.gov/) help translate the central ideas of the quality 
movement into school evaluation. 

The concept of systemic feedback on core processes is central to the quality movement. 
Feedback systems are essential for developing organizations that can learn from prior practices 
and intentionally shape practice to achieve anticipated ends (Senge, 1990; Senge et ah, 2000). 
Feedback processes translate organizational outputs into useful information that guides 
subsequent input behaviors. The lack of widely accepted output measures has been a primary 
obstacle to creating systemic feedback processes in schools, making it difficult to generate 
legitimate output information to be used as feedback. The recent press for accountability through 
standardized testing can be seen as an effort to provide legitimate output measures for schools. 
However, even with access to legitimate feedback information, feedback in human systems 
cannot be reduced to a mechanical procedure for incorporating appropriate information into 
organizational processes. Ilgen, Fisher, and Taylor (1979) emphasized the role of individual 
cognition and sense making by describing how organizational feedback must be perceived and 
accepted as valid and by emphasizing that recipients must be motivated to respond in ways that 
improve the process. Ironically, the more recipients are known for their ability to give advice and 
guide learning, the more willing they become to circumvent the implication of critical feedback 
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(e.g., Argyris, 1991). Nadler (1979) noted that the feedback process gets more complicated in 
group processes in which individuals struggle to understand the meaning of feedback for 
individual and organizational goals. 

Feedback systems require component functions to gather and provide opportunities to 
sort through relevant data, as well as structures to introduce findings back into the core 
organizational processes (Watson, 2005). However, the traditional loosely coupled structures of 
schools have worked against generating or integrating feedback about core instructional 
processes across classrooms in schools (Weick, 1995). Separating instructional from 
administrative practices effectively creates multiple information loops for organizational control 
and for teaching and learning within schools (Meyer & Rowan, 1983; Thorn, 2001). The 
traditional isolation of classroom teachers from each other further fragments the potential for a 
common vocabulary of information exchange, and the traditionally severe demarcation between 
teacher evaluation and teacher learning prevents school leaders from using organizational 
measures of student learning to help individual teachers improve their instructional practices. 
Because the feedback systems include multiple measures of organizational effectiveness, student 
learning, program design, and teaching practice, knitting these diverse kinds of knowledge 
together will tighten the coupling of leadership and teaching practice. 

DDIS Framework 

A DDIS builds a feedback system as a form of organizational capacity to meet the 
accountability demands for student learning. In building a DDIS, leaders use a variety of artifacts 
to reshape local accountability systems in order to satisfy the demands of external accountability 
systems. Our model DDIS includes six component functions: (a) data acquisition, (b) data 
reflection, (c) program alignment, (d) program design, (e) formative feedback, and (f) test 
preparation (see Figure 1). Leaders and teachers work to help these functions come together to 
translate student achievement data into useful classroom knowledge to inform teaching and 
learning. 

1. Data acquisition refers to processes designed to seek out, collect, and prepare information to 

guide teaching and learning. The primary data collected and organized is standardized 

student achievement test scores. However, many other types of information are required to 

guide teaching and learning, including: 

• Guidance information (student placement and behavioral records) 

• Student demographics 

• Classroom grades 

• Data on, and observation of, teacher personnel 

• Community survey data 

• Budgetary information 

• Master schedule and calendar information 

• Curricular information 

• Technological capacity 
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Figure 1. The data-driven instructional systems model. 












































Data storage (data warehousing) and reporting capabilities are also included in the data 
acquisition function. Relevant data are often housed in different organizational locations and 
in different formats, and streamlining data access can run up against political obstacles. Local 
data acquisition systems often build on existing district and state accountability systems. 
However, NCLB requirements for specific information on student performance have created 
a need for custom local data systems, and vendors have responded to this need with a variety 
of data storage and analysis products. 

2. Data reflection refers to processes developed to make sense of student learning data that 
result in goals for improving teaching and learning. While reflection is a constant and 
unpredictable consequence of working in data-rich environments, the data reflection 
component of the DDIS refers to structured opportunities for teachers and leaders to 
collaboratively make sense of data. These opportunities for structured sense making can be 
organized as whole-school, grade-level, or subject-area meetings. The duration of reflection 
sessions varies as well: some schools hold one-time whole-school meetings; others meet 
regularly throughout the school year. Successful data reflection involves problem framing 
and concludes with the determination of goals for a plan of action. 

3. Program alignment involves processes to make the school’s instructional program congruent 
with relevant content and performance standards and with the actual content taught in 
classrooms in order to improve learning and meet student needs. Program alignment 
activities take place at several levels. Alignment practices ensure that the school curriculum 
addresses content standards through grade-level and subject-area meetings. Alignment 
activities also include teacher and peer evaluation to ensure that what is in the curriculum is 
what is taught in classrooms. Finally, program alignment considers non-curricular 
initiatives — such as guidance and support programs, professional development, and 
community outreach — to make sure that existing school commitments address the concerns 
for which they were designed. Since school districts often provide alignment tools or 
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mandate standard curriculum for alignment purposes, the story of alignment involves school 
interaction with external agencies. Program alignment is a key aspect of planning and 
program evaluation. 

4. It is through program design that a school acts on perceived instructional needs by creating 
or adapting curricula, pedagogies, student service programs, and instructional strategies to 
improve student learning. Program design addresses the range of policies, programs, and 
procedures adopted or designed by the school to address recognized problems. Instructional 
program design can be addressed at the school level to meet the needs of groups of students 
through curriculum design, or it can be implemented through student-level programs that 
evaluate and determine the needs of individual students. The financial capacity to acquire and 
maintain programs, as measured by budgets and grants, is also an aspect of program design. 
Taken together, programs designed by a school can have varying degrees of coherence, as 
evidenced by the compatibility of internal characteristics or their perceived coherence by 
staff. 

5. Formative feedback produces learner-focused iterative evaluation cycles designed to create 
ongoing timely flows of information to improve both student learning and instructional 
program quality across the school. The generic notion of formative feedback is information 
useful for improving teaching in order to help the learner take ownership of learning. 
Formative feedback in a DDIS is intended to improve program design. Formative feedback 
can include information on student learning or teacher practice, but it becomes part of a 
DDIS when information about teacher practice is utilized in improving program design 
efforts. Formative feedback differs from data acquisition and reflection in that it refers 
specifically to information gathered about the programs put in place at the school. We 
hypothesize, however, that the distinction between data acquisition, data reflection, and 
formative feedback becomes increasingly blurred over time in a mature DDIS. 

6. Test preparation includes activities designed to motivate students and to develop strategies 
for improving their performance on state and district assessments. Test preparation can cover 
a range of topics, including test format, test-specific skills, test topics in which school/district 
students are demonstratively deficient, and habits shown to improve test scores (e.g., eating 
good meals). Although denounced by some researchers as an effort to “game” the testing 
system, test preparation is viewed by many schools as an extension of their program design 
efforts specifically intended to help make children comfortable with increasing testing time 
and pressure. 

Our concept of a DDIS in schools builds on recent proposals for how schools should use 
data to achieve results in student achievement (e.g., Bernhardt, 1998; Love, 2002; Schmoker, 
1999). These models offer guidelines for assessment, goal setting, analysis, implementation, and 
feedback. However, a distributed leadership perspective suggests that leaders should take an 
active role to link these components into structures through which information can flow from 
tests to teachers in a school. A distributed leadership perspective also suggests that leaders need 
to take account of the existing system of practice in building DDIS components. Schools already 
have artifacts that collect and distribute data; the task of leaders is as much to make sure these 
existing system components talk to each other as it is to design and implement new component 
artifacts. 
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We now turn to a discussion of the data we have collected over this past year to illustrate 
how the DDIS components function together to improve student learning in schools. 

DDIS in Action 
Overview 

Over the past year, we have engaged in a National Science Foundation-funded research 
project designed to study how leaders create systems to help teachers use data in their 
instruction. In the first year of our study, we focused on how DDIS components functioned in 
four Midwest elementary and middle schools identified by their improving test scores and their 
reputations for having leaders who use data well with teachers. After consulting with education 
leaders at university, state, and district levels, we selected eight schools recognized for strong 
leadership and data-driven decision making. We then narrowed our list to the four schools with 
the strongest records of improving student achievement (Table 1). 



Table 1 
DDIS Schools 



School^ 


Grades 


Location 


Size 

(students) 


% free/ 
reduced- 
price lunch 
(2004) 


Key artifacts 


Principal 

tenure 


Pearson 


K-6 


Rural 


300 


42% 


Guided Reading 


8 years 


Malcolm 


K-6 


Medium 

urban 


350 


68% 


Respect and 
Responsibility 
program 


6 years 


Harrison 


K-8 


Large 

urban 


800 


70% 


Direct Instruction 


3 years 


Walker 


6-8 


Suburban 


600 


3% 


MAP testing 


9 years 



We conducted structured interviews (Appendix A) with the principal and the formal and 
informal leaders as nominated by the principal and identified through our fieldwork. We 
observed and took field notes on faculty meetings, professional development sessions, data 
retreats, and other important events as identified by the staff. We collected artifacts such as 
school improvement plans, staffing charts, budgetary information, and parent/community 
handouts. To make sense of the more than 1,000 pages of field notes and artifacts collected, we 
used a qualitative data analysis program (NVivo) to code our data using an analytical scheme 



^ School and other names are pseudonyms. 
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based on the DDIS framework (Appendix B). After an initial pass through the data using the 
DDIS coding framework, we refined our schema to catch the nuances of the practices we 
observed. This first report on the data will help us test the validity of the DDIS and refine the 
model so that it better describes the practice of successful leaders. 

DDIS Functions in Schools 

We found clear evidence of the operation of each DDIS function in each school. Each 
school collected and used data; aligned, redesigned, and analyzed its instructional program; and 
provided test preparation guidance. However, we found wide variation in how the functions were 
developed in each school. Here, we consider the ways in which the four schools demonstrated 
the six DDIS functions in practice. 

Data Acquisition 

A central assumption of a DDIS is that data fuels the improvement process. The data 
acquired must have the potential to inform teaching and learning, but it need not be limited to 
test score results. As a district official told us: 

[T]he message that we give to schools is that they need to use student performance data from a 
variety of sources in order to put together the whole picture of student achievement. We don’t 
want them to focus solely on standardized test scores as they develop their educational plans, 
even though those are the most readily available because they are all electronic and the central 
office can neatly provide them with reports. 

While accountability demands encourage the reduction of data to simple test scores for reporting 
purposes, the school leaders in our study impressed upon us the importance of collecting a much 
wider variety of data to improve instruction. The teachers and administrators in the schools 
realized that improving learning requires a more sophisticated definition of data. As the same 
official expressed: 

We do want them to look at [test scores] and understand student performance as measured by 
standardized tests. We also want them to develop systems internally so that they can use 
performance data from the classroom in an ongoing way to monitor whether or not their kids are 
on course to meet objectives that they have in their educational plan. 

In the schools we studied, we found that data acquisition consisted of three subsidiary 
functions: (a) data collection, (b) data storage, and (c) data reporting. Below, we review these 
different aspects of the data acquisition process. 

Data collection. School leaders had access to standardized test scores both through paper 
reports sent to the schools and through online data portals maintained by the state test publishing 
company. In addition to summative test scores, each school collected information on attendance, 
student and community demographics, discipline referrals, expulsion and retention records, and 
student grades. The schools also collected a variety of information about the instructional 
program, such as student writing samples, teacher and class observations, survey-based climate 
data, and daily student assessments (usually in literacy). The district office played a central role 
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in data collection in three of our four schools. These offices collected district and state 
assessment information and made it available to principals online. 

The schools realized that data generated for administrative accountability purposes were 
insufficient to support local change processes. Each school had developed a number of internal 
data acquisition systems to guide instructional improvement. Harrison School, for example, has 
exploited the rigorous data collection system in its Direct Instruction (DI) program to generate 
data about teaching and learning. Harrison’s principal described how DI helped guide her data 
collection: 

[WJe’ve really become quite talented I think at data collection, particularly with the use of Direct 
Instruction, because [data collection] is an integral part of that program. The teachers . . . daily 
[are] collecting data on students, on how they’re performing, on how they’re doing. . . . We do 
check-outs, whether or not they pass tests after every five lessons in Direct Instruction, so all of 
that information gets collected, it gets compiled. 

Another Harrison staff member noted that, in addition to the data collected after every 
five lessons, the teachers maintain daily records of student learning: 

Well, there’s lots of data that teachers keep in terms of Direct Instruction. So there’s the weekly 
checkouts, and there’s the responses to the daily textbook and the daily workbook and all of that’s 
recorded and turned in weekly. 

As part of its data collection efforts. Walker School turned to the Northwest Evaluation 
Association’s Measures of Academic Progress (MAP) program, a computerized adaptive test that 
yields timely data on students’ progress toward meeting state achievement standards. Walker 
School’s MAP program is discussed further in the Formative Feedback section below. 

In addition to systems for recording student achievement data, our schools have 
developed systems for managing attendance and discipline data. One associate principal 
articulated the school’s method of recording attendance data as well as the connection between 
attendance and learning: 

There’s a log that we keep in the attendance room and we look at about every 2 weeks or so to 
see if we have any updates, and whenever we have a situation, we log it into this record. Our 
student management record system is what records student attendance and grades, and I review 
weekly reports of how we’re doing with student attendance because when we’re having students 
coming into class late, it is a disruption to the teaching and learning that should be expected to 
begin immediately once the class gets under way. 

Malcolm School developed a robust system for recording student discipline data to 
address the influence of student behavior and school environment on teaching and learning. 

Their Respect and Responsibility (R&R) program used a spreadsheet to record and track 
behavioral data. The school’s principal reviewed the data during a weekly meeting with the 
administrative team, and at any other time as needed. The R&R program is an example of a 
sophisticated local data collection system that does not include test scores and that serves as the 
starting point for a DDIS addressing student behavior. 
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Data storage. Once data are collected, they must be made accessible to leaders and 
teachers. We observed a range of low- and high-tech data storage practices, ranging from filing 
cabinets and binders to sophisticated off-site data warehouses. Three of our schools relied on a 
significant district investment in data warehousing technology for data storage purposes. Still, 
each of the four school principals relied primarily on low-tech printouts and notebooks to keep 
track of data relevant to the school program. Aside from standardized attendance and budgeting 
programs, schools’ local data storage systems generally operated independently of district data 
systems. The mismatch between high-tech district storage and low-tech local collection and 
storage meant that (a) local leaders needed to be literate in both systems and (b) principals acted 
as data conduits between the district and the schools. One district leader proposed a bridging 
system: 



[W]hat I think would be really eool is if we had one super application and achievement 
management system that would pull them all together — that would tie the curriculum that 
teachers deliver daily to the district’s learning targets and the state’s standards to performance on 
state and local assessments so that the full picture is there. You could put classroom assessment 
performance in there. You would have standardized test performance information. You would 
have curriculum that you delivered, and it would really promote this alignment of curriculum, 
instruction, and assessment that we’re advocating at the central office, but we don’t have a good 
vehicle for supporting that at the school level. 

The fully integrated data system of which some leaders dream is far from being achieved, 
although some have recognized its value and have made some degree of progress. However, 
even though principals talked about the existence of dual information systems, complaints about 
the problems of such systems came mainly from the district level. This may be because local 
school leaders realized that control over information storage determined the kinds of information 
schools could legitimately collect and that integrating all relevant information into a single 
system would likely reduce the autonomy and flexibility of local actors to acquire the 
information necessary to address emergent problems and increase external control over local 
schools. 

Data reporting. What data are reported depends on the kinds of information desired by 
school leaders to improve instruction. Thus, the kinds of reports requested by leaders and 
teachers provide a good reflection of the problems they perceive as worthy of addressing. In our 
observations, we found that data reporting was often an interactive, dynamic process. In low-tech 
districts, data reporting occurred via interpersonal interactions with people in charge of data 
storage. In Pearson School, for example, the principal and several lead teachers met regularly to 
develop reports on student learning collected through regular testing and anecdotal information. 
The discussions surrounding report generation and results provided an occasion to develop 
shared understanding of purpose and strong professional community among the leadership team. 
Another school developed what they called a Critical Index in order to report weekly data. An 
administrator at the school collected the data and compiled the Critical Index every month to 
expedite its review by the principal. The principal explained the origin of the Critical Index and 
the recognition it has since received: 

[OJther schools have used our Critical Index, and they’ve used some of the other sheets that 
we’ve used to collect data, because they’ve used it as an example. And that was really developed 
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by our needs, and looking at the needs of what do we need to find out for information that will 
help us improve, and then how can we develop this form? 

In high-tech districts, data reporting was guided by district-designed query templates 
accessible online. One district assembled hundreds of online report templates for principals and 
trained them in accessing the system and determining which reports to use. These online tools 
could be used to generate reports using customized search terms, including demographic limiters, 
academic subject or standard, and year. In some districts, principals also had liaisons in the 
central office to help procure data reports. The reports were then used to structure conversations 
with staff. The customized searches narrowed the data to only those issues the principal hoped to 
know about at the moment. The “data reduction” function served by district reports can be very 
important for principals who can feel, as one admitted to us, “somewhat overwhelmed with 
data.” 

Data Reflection 

Making sense of data is a critical function of a DDIS. Feldman (1989) described sense 
making as an opportunity “for organizational members to understand and to share understandings 
about such features of the organization as what it is about, what it does well and poorly, what the 
problems it faces are, and how it should solve them” (p. 19). Data reflection occurs through 
structures leaders build to help their schools make sense of which problems to address and set 
goals for the instructional program. These occasions for reflection can take place at the district, 
school, grade, or content area level. We found two kinds of data reflection in our schools: (a) 
data retreats and (b) local data reflection activities. Schools typically engaged in one or two 
whole-school and/or district data retreats sponsored by the district or another outside agency and 
provided multiple structured opportunities to reflect on data over the course of the school year 
during regularly scheduled meetings. 

Data retreats. Data retreats provide the opportunity for large groups within districts 
and/or schools to make sense of achievement data and to set organizational goals. Data retreats 
require schools to assemble a variety of data, to discern patterns in what they have assembled, 
and then to generate hypotheses and action plans to address these concerns (Sargent, personal 
communication, March 10, 2005). Two of our schools engaged in district-led data retreats just 
after the close of the school year. Attendance was voluntary, although one district arranged for 
college credits to encourage teachers to attend. The Walker district, for example, used its data 
retreat to tie data reflection practices into their annual “visioning process.” The district 
administrator explained how, once they identified an issue, they examined more data to 
determine contributing factors. This “deeper” reflection, as he called it, came only after the 
initial recognition of the problem: 

We also dug deeper and said ok, of those students who are performing at the lowest levels, what 
kind of classes are they taking? We found out that the boys are taking lower level classes and 
they’re taking fewer core academic classes than the girls. 

Collaborative reflection on the data helped them narrow the problem of student achievement to 
the issue of gender equity and led to discussions about how to resolve the issue through the 
instructional program. 
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We found that retreat leaders were quite directive in their efforts to lead reflection. 
Tension can exist between such directed agendas and open exploration of the data, between 
retreats run as a forum for announcing results and those run as a forum for solving problems. 

One district leader explained how they tried to resolve this tension at the district data retreat: 

Our philosophy is not to do the PowerPoint and then say, “Okay guys, get to work.” Or come 
over and do the work for them, you know? We feel as though it’s more powerful if we give them 
what the task is and then let the teachers and principals at the building level dig into it and have 
their own conversations. We may ask leading questions, [but] we want them to have the “ah-ha!” 
themselves. 

A district administrator shared his observations of how different principals in his district 
led subsequent school reflections: 

One of the elementary principals . . . allowed the staff members to lead as much of the discussion 
as possible. ... I think he was the most successful. The ones where the principals pretty much ran 
the show, they ran everything. Once [the teachers] got into their small groups, [they] were the 
most dysfunctional. . . . The teachers felt a disconnect, didn’t feel a part of the results. There’s a 
fine line between having your principal lead and having the teachers help themselves. 

The desired outcome of a data retreat is the development of district goals for teaching and 
learning. One Walker district administrator explained how their goals developed: 

We found these positives, we found these negatives, you know. As a result of that, they take that 
information and then they review it and then that’s the first thing that you need — just to lay all of 
your ideas out and things out and you have all of that information over the summer and then the 
next time we meet, we narrow down — I use different methods each year to narrow it down. So we 
narrow down what we feel is critical goals for the district, and that’s basically what we do, and 
we end up with four, or maybe five at the most, district goals because I think if we get over that 
many, you’re never going to achieve them. 

The departments and schools in the Walker district were then expected to derive their local goals 
from the district’s overall goals. These goals had to be measurable, attainable, and relevant, and 
they were subsequently reviewed by the board of education: 

[E]ach department will have a set of goals that are in a lot of cases related to the assessment. . . . 
50% in each math quartile will meet or exceed an individual growth target by evidence of the fall 
and spring performance. ... So data is driving one of the goals in that building — and you can 
look through the rest of this and find the same thing. . . . [In] social studies . . . they’re talking 
about areas of concern on the [state exam] and grade point averages. So they’re looking at data to 
make decisions about improving instruction and that’s their goal for the year that they’re going to 
be targeting — certain things that really relate to assessments that they do and that we do. 

Local data reflection activities. Each of our schools structured a number of occasions to 
develop local goals, as well as to plan how to meet the goals formulated at district data retreats. 
On average, we observed four data reflection sessions during the year in each school. These 
reflection sessions addressed data disaggregation, item analysis, and individual student progress. 
Data disaggregation activities at the school level began with breaking achievement data down to 
the individual student level, which then allowed teachers to relate additional behavioral and 
social data to an understanding of the achievement results. We found the overwhelming majority 
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of discussions about achievement data focused on helping struggling students achieve 
proficiency; few discussions concerned how to raise student achievement from proficient to 
advanced levels. We also found that teachers were more engaged in discussions about individual 
students than in consideration of grade-level or subject matter groups. In each of the schools, the 
formal leader took responsibility for shifting discussions from individual student interventions to 
programmatic implications. 

The school-level reflection sessions relied on simple graphs and tables of achievement 
data. The value of data reflection seemed to lie not in the sophistication of the analysis, but in the 
frank discussion of practice and in staff ability to identify areas for future effort. Local data 
reflection sessions involved different groups of staff members at different schools. Pearson 
School, for example, used professional development time for “implementation days” that 
involved the whole faculty in discussing student achievement and student behavior reports. In 
addition, the Pearson leaders helped organize separate days for the fifth-grade teachers, for 
example, to reflect on their students’ achievement scores, and voluntary sessions at the beginning 
and end of the year dedicated to setting and evaluating the school learning plan based on 
achievement data. The inclusion of special education staff in all Pearson data discussions seemed 
to reflect a strong connection with classroom teachers, enabling the staff to discuss the needs of 
struggling students together. 

Other principals took different paths to develop their schools’ capacity to reflect on data. 
The Harrison School principal, for example, described how she had developed an increasingly 
collaborative system for data reflection that started with a small group of trusted colleagues and 
then expanded to include more staff members: 

I did this a couple of years, and my school psychologist did this a couple years with us and with 
the program implementer — you know, the three of us would sit down and actually do some item 
analysis and pull it out and extrapolate it for the staff and hand it over. That’s what we did like 4 
years ago. . . . [B]ut now we’re more focused on having the committees do it so that everybody’s 
involved in it, so that’s been the push. 

The Malcolm School principal convened her administrative team on a weekly basis to review the 
school’s behavior data reports. She then used these meetings and reports to identify emergent 
issues in the school and to develop strategies to share the academic and behavior data with 
teachers during regularly scheduled grade-level meetings. She shared these data to create a 
“shared culture around children” and establish “collaborative conversations around kids,” 
particularly to identify “what’s going on or not going on with the kids.” After one such meeting, 
she described the discussions that resulted from sharing data with her staff as “the beauty of 
data — we can have these conversations.” 

Taken by themselves, data reflection activities are opportunities for staff to understand 
achievement patterns in their schools. When integrated into a DDIS, however, data reflection 
activities are powerful staging areas for school leaders and teachers to frame the problems they 
seek to solve through program alignment and design and to develop goals that can be measured 
in terms of student learning. Data reflection activities in a DDIS help leaders narrow problems to 
make them solvable. Without identifying and defining the problems in a deliberate manner, the 
schools can only hope that their efforts will actually lead to substantive improvement by virtue of 
good fortune, and our schools cannot afford to bank on fortune alone. 
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Program Alignment 

The program alignment function served two key purposes in the schools we studied. 

First, alignment activities served a problem-finding purpose by pointing to areas where the 
current instructional program did not address student learning needs. Second, alignment helped 
the schools understand the degree to which their current programs fit together and addressed 
relevant content and curricular standards. We also found the function of program alignment 
stretched our view of what is encompassed in the instructional system. In implementing the 
alignment function, school leaders looked not only at curricula and classroom materials, but also 
at student services, budgeting, personnel, and assessment practices. 

To illustrate the dimensions of program alignment, we highlight Harrison School’s 
decision to adopt Direct Instruction as a school-wide instructional program. The principal at 
Harrison described how alignment activities helped her begin to diagnose problems of student 
learning soon after she arrived at her school: 

So, I started the discussion when I first got here with some of the staff in the fall ... in regard to 
where we needed to go with student achievement and with curriculum. Because the one thing that 
I did notice was that everybody was doing their own thing — the former principal was not aware 
of a specific reading program, for example, our reading scores were quite low, and our reading 
resource teacher wasn’t aware of a specific program either ... I mean wasn’t aware of what 
programs all of the teachers were using. 

In this case, the principal perceived that alignment activities were a condition for creating a 
system to improve student learning. She used alignment activities to understand, and to help the 
staff see, how the current program fit together in order to determine an action agenda. She 
described her process for investigating what the school currently offered: 

So I had the reading resource teacher survey the staff, and they were basically all doing different 
things. There was no consistency within the building. We had wonderful staff, great teachers, 
hard-working individuals, lovely school, great environment, caring community, but there wasn’t 
any consistency with curriculum. 

At this point, the principal could have pursued several paths toward building a more coherent 
instructional program. Ultimately, she chose to investigate how a comprehensive curricular 
program could bring coherence: 

We did an investigation. ... I looked for volunteers to go out and look at programs in other 
schools, go out and do some research, go online, speak to parents — we had parents involved in 
that — and we basically developed an investigative committee, and we went out to various 
schools. They came back with a great interest in the Direct Instruction curriculum because they 
saw it in another school. I had known about it because I came from a DI school, and that was one 
of the schools that I had sent them to so they could see the great things that can happen with that 
curriculum. But they were actually the ones that saw it, and they were the ones that wanted to 
start the program. 

A key aspect of DI is that it provides direct, ongoing measurement of student learning in 
relation to curricular goals. The decision to adopt DI at Harrison provided a powerful occasion 
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for sharing practice, engaging in collaborative professional development, and building a budding 
professional community among Harrison teachers. 

We found that the principal and staff at Harrison did not see DI as a one-stop solution for 
student learning issues. Rather, the staff continued to engage in problem- finding activities within 
the constraints of the DI program. The decision to adopt DI helped to constrain, but did not 
solve, the student achievement issue at Harrison. This case illustrates the second purpose of 
alignment — determining the congruence between the instructional program and relevant learning 
standards. With each passing year, the Harrison staff became increasingly proficient with the 
curriculum, but they found that not every child was successful on the standardized tests. The 
Harrison principal explained how, in an effort to meet the needs of all children, the staff began to 
consider the alignment of DI with learning standards: 

In the third year [of DI], we were really focusing on, okay, where are the gaps .... So we’re 
growing each year as we’re learning more and being better at wbat we’re doing. It’s been easier 
and easier each year in regards to understanding how to align it and what materials we may 
need .... Other literature, other books that we bring in, other resources that staff will bring in or 
teachers will recommend, we’ll purchase or provide for staff, we’ll develop the supplemental 
material library for staff, so lots of different resources. 

The staff also engaged in ongoing alignment activities to determine where the DI program 
needed to be supplemented to meet the needs of particular student groups: 

[F]or example, . . . with our Hmong refugees ... we have . . . students that don’t know any of the 
language or speaking or understanding anything about America, let alone being able to say, 
“Hello” during the day. One of the things that we’ve been able to do is take a look at some of the 
specific needs and supplement with some technology. So we’ve been able to provide some 
programs, too, in that area. We’ve been able to supplement for students’ comprehension with, 
like, Accelerated Reader, too — we use that — we use Math Facts in a Flash, we use some other 
technology programs to assist in helping with some of the gaps that we find in the regular 
curriculum, so that’s been supportive. 

In Malcolm School, we observed how alignment activities focused on curriculum and 
state assessments. This school’s district assumed that if curriculum was aligned with state 
standards, students would learn what they needed to know to excel on the state exam. The 
director of curriculum and instruction within the district described his role in the realignment 
process as follows: 

[AJfter the tests are completed, after the results have returned, then my role pretty much kicks in, 
and I do things like [work with the] staff [to] do item analysis of the . . . test scores. We make 
sure that we’re teaching the right curriculum, that they align their curriculum with the state, and 
in some cases, there is a state standard established by the state associations, and so they need to 
align their curriculums with those components. I make sure that that’s fine. 

Whereas the district alignment process focuses on how to understand test scores in terms 
of the state test, school alignment efforts focus on where and how the tested topics are covered in 
the curriculum. One principal explained: 
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The state sends us an item analysis, it’s kind of a gross overview, and then what I ask the teachers 
to do is to dig into that deeper and pull out segments so that they can correlate it more tightly with 
the curriculum. ... You look at something, and you say, “oh, let’s go hack and look at the 
curriculum, what are we spending too much time on, what are we not spending enough time on, 
let’s start to dissect this, and let’s see how we can approach it differently, and then let’s jack it up 
to the next layer and let’s look at our instructional practices.” 

Here, we can see how the line between data reflection and program alignment begins to 
blur in activities such as item analysis of standardized tests. Principal and teachers use the test 
items to reflect on student achievement scores in relation to what teachers report covering in 
their classrooms, thus integrating the reflection and alignment functions into a single activity. 
Pearson School dedicated 3 full in-service days per year — 75% of their annual professional 
development time — to sessions for teachers to disaggregate data and engage in item analysis of 
the state exams. However, the local expertise required to successfully engage in item analysis is 
expensive to develop. The Malcolm principal described how the capacity to engage in this kind 
of reflection/alignment activity was just beginning to emerge among her teachers: 

I would say, if I can he very honest, it’s a real challenge, we prohahly, we are now — and granted 
this is only the 2“^^ year — we’ve really gotten any worthwhile information hack ... I would 
mayhe think I have three people on staff that really are getting an understanding of what that 
really means, what does it mean to look at this question that 75% of the students got wrong, and 
what was the content of that, and what was the standard behind that . . . [Y]ou find a teacher 
who’s doing it well, and you make a hig deal about that teacher doing well — that “Oh, you know 
what, he’s really got a good idea here, here’s what he kind of came up with for his chart, and go 
on ask him to talk about if’ — you know you do that kind of thing, you kind of try to get teachers 
to teach teachers, because that is by far and away the most effective thing. Now that we’ve had 2 
years of experience . . . teachers are getting a better idea. 

A final point concerns the alignment of the instructional program with student services, 
community outreach, and other programs. All the schools we studied had sophisticated special 
education placement services to coordinate instructional program services with individual 
student needs. Other schools have moved beyond typical special education services to develop 
extensive student behavior programs (e.g., Malcolm’s R&R program) to coordinate student 
support services with the demands of the instructional program. Still other schools have 
developed programs to align instructional offerings with community ESL or adult literacy needs. 

Program alignment activities in themselves provide schools with information about the 
range and organization of their current programs. Integrated into a DDIS, however, alignment 
activities come to life as a “reality-testing” function to inform goals set through data reflection. 
Building an information flow within schools requires that leaders align programs to determine 
where resources have already been spent and where they will need to be allocated. 



Program Design 

Leaders and teachers use a variety of artifacts to assemble the instructional program of 
their schools. While some artifacts are designed by a school to meet specific instructional needs, 
others are inherited, and still others come into the school from the district or state (Halverson, 
2003). We found two distinct categories of artifacts used to shape school instructional programs: 
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(a) curriculum-based programs, used to instruct groups of students in conventional classroom 
settings; and (b) student-based programs, designed to customize institutional resources to the 
needs of individual students. We discuss each of these below, along with another aspect of 
program design — the acquisition of additional resources to fund new programs. 

Curriculum-based programs. Curriculum-based programs include the variety of 
programs schools use to guide classroom instruction. For the purpose of the DDIS, we found that 
each of our schools had selected a variety of reading, writing, and math curricula to meet the 
state accountability demands in these subjects. The literacy curricula ranged from comprehensive 
programs such as Direct Instruction to balanced literacy approaches that required teachers and 
leaders to customize phonics and whole language approaches to the needs of their students. In 
math, schools followed their district’s lead in adopting comprehensive curricular packages such 
as Everyday Math. All of the schools felt that the comprehensive approach to math had 
significantly improved teacher capacity and learning, and consequently math had received less 
attention in the DDIS discussion. Improving literacy scores, on the other hand, continued to 
prove a stubborn challenge and served as the central topic of data reflection and program design 
in the schools’ DDIS. Program design in science and social studies was left largely outside the 
purview of the DDIS, in part because these subjects did not receive the same emphasis from the 
state accountability system. At Pearson School, for example, social studies and science curricula 
were guided by a multiple intelligence-based curriculum design process largely outside the 
DDIS. Groups of teachers met monthly by grade level, using a lesson study-like format to 
collaboratively design and discuss sample social studies and science lessons. These design 
efforts were reported at faculty meetings, but their results were not discussed with the same 
attention as the school’s literacy efforts. 

Program design was also used to provide targeted solutions to specific problems that 
surfaced through the DDIS. To illustrate the use of program design within the DDIS, we consider 
the reaction of Walker School to the revelation that they had failed to meet the 2004-05 NCLB 
adequate yearly progress (AYP) requirements for special education students in reading. Failure 
to meet AYP was seen as a public black eye for a suburban district widely recognized for 
educational excellence. A district curriculum specialist reflected on how he attempted to analyze 
the achievement problem in special education: 

A year ago I thought we were in pretty good shape with special ed reading ... on the [state 
exam], and then we dropped a couple points this year [and] didn’t make AYP. . . . And last year 
[2003], our special ed students were ahead of the district standard [61% of students testing at 
proficient or advanced on the state exam] .... This year, we’re at 57%, so we didn’t make it by 
shooting towards that mark. So then we looked at the state, and we had to decrease our lowest 
group by 10 percentage points. . . . But ... as you know, our special ed students can read; we take 
a look at our students at large, and 95% of our kids in the district were at proficient or advanced 
reading. And that’s pretty good, you know, so, we’re not sure what to make of it. . . . Our test 
participation rate was at 100%, I think. Our graduation rate was at 95%. . . . It just doesn’t make 
sense to me. 

Except for reading in special education, the specialist pointed out how the overall district scores 
were well above the state average. Even in special education, he reasoned that most of the 
children were proficient in reading according to other district measures. 
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Ultimately, the Walker principal and district leaders addressed the problem by finding an 
intervention program to help raise the achievement of the low-performing students by 10% in 
order to meet the AYP. A district official proposed Read 180, a pull-out program designed to 
offer intensive reading remediation through a combination of classroom and computer-aided 
instruction: 

We’re looking at a descriptive reading program for low-achieving students called Read 180. . . . 
We found out from our last few [state exams] that our low verbal students are not performing to 
their level of aptitude in reading. So we’re looking at having a reading specialist or a classroom 
teacher or a special ed teacher work with the very lowest score readers. Read 180... has 90 
minutes of reading everyday. . . . We’re looking to try it [as] a result of looking at our verbal 
readers’ not progressing the way that we would want. 

After a thorough investigation of the program and its potential for improving the reading of 
special education students, the district decided to purchase the program. By January, teachers 
were trained to use the program, and students were placed in a pilot program. The principal and 
staff used achievement data to help place students into a pilot trial period. The Walker principal 
explained: 

We also made a very conscious decision of the kids that we’re going to target for the pilot. We 
used data. ... I created a template or a table, that took the Lexile levels from the kid’s most recent 
math test. I took their reading and writing scores, and I took their reading score percentile on the 
MAP, but where are they in the whole big picture thing. . . . Then I took their language use 
percentile, I documented the types of support [the students received] . Then I looked at their 
schedules to determine what would be the least disruptive method of pulling them [out for Read 
180]. . . . [We] took the kids that . . . would need this the most. 

The above example illustrates DDIS program design. Selecting programs to address 
emergent student achievement issues is certainly not new in schools. The Walker example stands 
in contrast, however, to the pattern of schools’ adopting a number of incompatible programs, 
resulting in incoherent systems of practice, or what Bryk, Sebring, Kerbow, Rollow, and Easton 
(1996) called the “Christmas tree” phenomenon. Integrating program design into a DDIS helps 
schools use student achievement data and existing program commitments as a check for program 
bloat. The Walker example shows how achievement data were used to determine program 
weakness and to target just the right artifact to address the problem. 

Student-based programs. Student-based program design approaches the instructional 
question from the perspective of individual, rather than collective, student need. Drawing on the 
powerful precedent of the special education individualized education plan (lEP), student-based 
program design creates systems to develop customized instructional plans for struggling 
students, while avoiding the stigma that often results from special education identification. 

In the schools we studied, the goal of student-based design reflected how school leaders 
and teachers perceived the problem to be solved in student achievement. Eeaders at Malcolm 
School, for example, felt that student well-being was a critical aspect of improving student 
learning. Eeaders reasoned that students unable to cope with the restrictions of the classroom 
environment would have a much lower chance of experiencing success in the instructional 
program. The Malcolm Respect and Responsibility (R&R) program was designed as a social 
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problem-solving tool, not a discipline tool. A teacher involved in the R&R design remarked, “we 
might hand out a little discipline [in R&R], a little consequence for the social problem-solving 
issue, but discipline still comes from the principal.” 

The R&R program grew out of a comprehensive school audit intended to identify the 
problems Malcolm School was encountering in the initial year of the current school principal. 

The audit resulted in the creation of four problem-solving teams, including one dedicated to 
“climate and order.” This team proposed both curriculum-based solutions (life-skills lessons built 
into the normal instructional program) and R&R, a student-based program that described a set of 
escalating steps for intervention with a disruptive student. The R&R team consisted of the school 
psychologist, the social worker, an educational assistant, a school facilitator, and the school’s 
assistant principal. A team member was on duty throughout the school day. When a student 
behaved in a manner that compromised his or her learning or that of classmates, the teacher was 
expected to intervene. If the intervention was unsuccessful, the R&R team member on duty was 
called in. The R&R team member helped the student solve the problem with the skills necessary 
to return to class. Chronic cases of misbehavior resulted in an intervention process crafted by the 
R&R team in consultation with the parents and classroom teachers. This intermediate 
intervention step helped make the behavior problems of a number of students manageable 
without special education identification. 

From a DDIS perspective, the key aspect of R&R was the wide range of student data both 
generated and analyzed by the staff. All referrals were documented, tallied, and analyzed in 
weekly meetings by teams of school administration, staff, and teachers. The focus of the analysis 
was to make sense of the frequency, severity, and patterns of behavior as indicated by the R&R 
referral system. Both cumulative and student-level behavioral data were integrated into the data 
reflection process at Malcolm to help teachers move beyond test scores and get a handle on the 
education of the whole student. The effect of the R&R program was to provide the Malcolm 
DDIS with the information necessary to help teachers make the right adjustments in the 
classroom: 

Every room in this building [has children] with incredible issues — enough to scare people. But 
kids still learn, they come and they produce, they have good test scores. Teachers and kids are 
doing what they need to do in order to teach and learn. 

Acquisition of additional resources. A final aspect of program design is the acquisition 
of additional resources for necessary innovations. Each school we studied had an established 
record of grant acquisition. However, grants come with their own program demands that are 
often at odds with existing program requirements. The Harrison principal, for example, described 
how taxing a recently acquired federal Reading First grant had been for the school: 

It takes a lot of time. . . . There are many additional testing requirements with the grant. [It] takes 
about an hour to an hour and half to test each child. This means that teachers could possibly be 
out 8 days for the year just to assess their students twice. Because of the time required, I have 
worked with the district to have retired teachers come out and help test so that not so much 
classroom instruction is lost. This has helped, but it has been difficult. We are hoping that we can 
get assistance in reducing the assessment. [The] assistant principal also stated he hoped for 
changes to help reduce the cost of the assessment. The extra people are costing money. 
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Summary. The key feature of DDIS program design in each of our schools was the staff 
willingness and capacity to change core instructional programs in light of data analysis and 
reflection. These schools blended curricular and student-based interventions to customize 
instructional programs to adequately support student learning. Their experience leads us to 
believe that exclusive attention to either curriculum-based or student-based interventions can 
create a dysfunctional DDIS in which information does not flow across functions. Simply adding 
even the most rigorously tested curriculum-based programs can set up false hopes and failed 
expectations for student success, while leaving the real problem — the inadequacy of the existing 
instructional program — unaddressed. Similarly, student-focused interventions can lead to widely 
reported efforts to “game” accountability systems by focusing resources on the students on the 
verge of success while ignoring the needs of students on either side of the accountability 
threshold. Under the leadership of savvy administrators, however, accountability pressures can 
result in balanced approaches to program design that can benefit the learning needs of most 
students. 

Formative Feedback 

The formative feedback function is perhaps the most critical aspect of a successful 
DDIS — and the most difficult to implement. Formative feedback structures collect and create 
occasions to discuss the ongoing information generated by teachers and students about the 
quality of school initiatives. Most schools find it too expensive to develop systematic feedback 
on current programs, relying instead on occasional, anecdotal assessments. None of our schools 
demonstrated the capacity to provide systematic feedback on student learning across their 
instructional programs. However, the schools did provide examples of how feedback structures 
can be developed in specific areas. These examples fall into two categories: (a) artifacts 
developed by local school leaders and teachers; and (b) artifacts developed or purchased at the 
district level and adapted by local leaders to address school needs. We discuss both of these 
below, along with tools for providing formative feedback to teachers. 

Locally developed artifacts. The Title Fliteracy feedback program at Pearson School 
provided an excellent example of a system designed by local school leaders to provide 
systematic feedback on program initiatives. Pearson School was in a rural district with limited 
resources that forced the district to stretch personnel across schools. (The Pearson principal, for 
example, was also the principal of five smaller rural schools.) The Pearson leadership team had 
to rely on repurposing in-house expertise to provide feedback on program design. The principal 
worked with the Title I teacher to develop a sophisticated system to measure the effect of literacy 
program design on student learning. The Title I teacher, a veteran reading specialist with training 
in the Reading Recovery program, worked with teachers for 6 years to redesign the Pearson K-2 
reading program. The cornerstone of the program was Guided Reading (GR), a program that 
helps early readers develop effective strategies for processing text at increasing levels of 
difficulty (Fountas & Pinnell, 1996). GR relies on running records — individualized, ongoing 
formative student assessments — to help teachers organize groups for reading activities. The 
Pearson Title I teacher organized her schedule to spend time working with groups of students 
and teachers in each classroom in order to get a sense of teachers’ practice and student 
performance. She began assembling binders of running records information to track student 
progress over time, and she worked with teachers to supplement the GR assessments with 
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formative feedback tools from Reading Recovery and other programs such as the district- 
supplied Developmental Reading Assessment (DRA). The Title I teacher commented: 

Collecting data on how much kids understand and where their understanding breaks down helps 
us understand where they need more help. This gives our kids a chance to feel comfortable with 
what they are doing so they can say “I am a reader.” 

Taken together, these data provided a powerful resource for measuring program quality. 
Still, the data alone would not constitute formative feedback unless teachers used the information 
in their instruction. Pearson’s leaders realized the value of structured opportunities for reflection 
in making formative data useful. The Title I teacher described meeting weekly with every 
teacher and monthly with the K-2 and special education teachers to discuss and disaggregate the 
data: 



Data disaggregation time lets us discuss our children and our program. [In our discussions,] we 
also look at home life — we know that when families go through a divorce, children lose a year of 
academic progress. We also look at the data [during] midyear meetings and during our [school- 
wide] implementation days. 

Professional time dedicated to data discussion helps develop a strong professional 
community around literacy instruction and identify problems with the existing program. For 
example, in the early days of the Pearson reading program, teachers were unknowingly using 
different vocabulary to describe the GR program to children: 

When we started, we found that one teacher was saying “sound it out,” another teacher was 
saying “stretch it out,” another said “say it slowly.” Our top kids were figuring it out, but tbe kids 
who were struggling were saying “I gotta do something else?” They didn’t get it. . . . We have a 
common language now. Once they established a common language, the struggling readers could 
figure out that the skills sought by different teachers were the same. 

This complex system of formative measures served several key functions in the Pearson 
instructional program. First, it helped Pearson staff develop a sense of shared ownership of 
transformed practice. Although K-2 teachers continued to work in classrooms, they felt more 
connected to each other’s practice as a result of participating in the GR assessment system. 
Second, this professional community helped staff to use the formative feedback as an effective 
measure of program design. When teachers began to realize that GR was not addressing the 
needs of several students, one teacher shared her experience at an Orton- Gillingham phonics- 
based program workshop. After several other teachers attended the workshop, the Pearson team 
began to integrate Orton- Gillingham activities and assessments into the literacy program for 
selected students. Finally, the formative assessment program helped staff anticipate the results of 
the state exam. The Title I teacher described how she was “rarely surprised, because the running 
records help to determine where the children should be on the DRAs, which predict the [state 
exams] well.” 

Adaptation of received artifacts. Other DDIS schools relied on district-initiated programs 
to develop similar feedback systems for early reading initiatives. For example, the Malcolm 
School worked with the district’s Gateways to Literacy program, which also relied on teachers’ 
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keeping running records, to provide feedback for its balanced literacy initiative. A Malcolm 
teacher described the effect of the program on the school: 

So it’s no longer okay to say . . . “this child is just slow and struggling.” Really taking the 
responsibility and saying “I need to figure out what I can do differently because obviously what 
I’m doing isn’t working.” 

Malcolm teachers found the data provided by the Gateways program valuable for guiding 
their instruction: 

A teacher had a whole [first-grade] reading group stuck on a pretty beginning level. . . . They 
weren’t making progress, so we looked at the [teacher’s] running [records] and we realized that 
several of the kids in that group were decoding — they were trying to make sense, but they didn’t 
know the high-frequency words that most kids just know like this .... So we designed a new 
program to get those kids more practice with high-frequency words, and it was because the 
teacher had running records of that and we could actually see that it in their writing samples, too. 

The Gateways project helped this teacher use formative data to pinpoint student struggles and 
design a new approach that would better serve those students. 

Harrison’s Direct Instruction program also generated formative data through brief literacy 
assessments such as DIBELS (Dynamic Indicators of Basic Early Eiteracy Skills). The structure 
of the DI program itself provides a framework for measuring how well the program is being 
implemented. The Harrison principal described the tools provided by DI to keep track of 
program progress: 

The DI chart [helps us collect] the data weekly, compile it, and put together a monthly Critical 
Index [that] I look at, typically. I don’t look at the weekly data myself as much . . . unless there’s 
a thing that’s alarming or I need to address a teacher ... or if something wasn’t completed. . . . 
Like right now, I have a real concern about our Language for Learning in tbe K-4 classes, 
because they’re not progressing as quickly as they should be; they’re not learning what they need 
to be right now. 

The MAP program at Walker School served a similar function of providing ongoing 
information on student achievement to inform program design. The MAP tests generate 
immediate reports about how students are progressing toward state standards across subject 
areas. However, leaders must create opportunities for teachers to reflect on these data in order to 
make the information useful. Blink (2005) described how one district used MAP data to structure 
professional development and program design sessions across schools in literacy and math. The 
Walker leaders are currently developing structured opportunities for teachers to use MAP data in 
similar ways. 

Formative feedback for teachers. The tools described above are used to generate 
information about program success in increasing student learning. There are several other tools 
available to school leaders to build formative feedback for teachers into the DDIS. One obvious 
path is to link teacher evaluation into the formative feedback process. Unfortunately, the current 
conflation of formative and summative functions makes it difficult to untangle these functions in 
practice. Nevertheless, the schools we studied developed several methods for providing 
formative feedback to teachers. Each school invested in instructional coaches to provide 
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formative feedback to teachers, while leaving summative evaluation to the school principal. 
Harrison’s DI program used a literacy coach to help teachers adjust practices in light of 
formative data. The Harrison principal described the literacy coach’s value in assembling and 
distributing data: 

Having a literacy coach is really good . . . she’s the one person, quite honestly, where all these 
worksheets go. If there’s any red flags, [she brings it] to my attention, and I’ll address it, or then 
on a monthly basis it’s put together as far as an overall [picture of] how they’re improving, where 
they’re going, are they making enough gains. 

In addition. Walker School leaders encouraged teachers to develop action research projects to 
generate feedback about program initiatives as a path for teacher professional development. 

Summary. Formative feedback structures are both pervasive and difficult to identify in 
most schools. Leaders need to build such structures to generate information for teachers to adjust 
their instruction to improve student learning. Even though these practices are expensive and 
difficult to institutionalize, the net effect of successfully implemented formative feedback 
structures is to transform classroom practice by tightening the coupling between administration 
and teaching and showing how information loops can inform as well as evaluate practice. 

Test Preparation 

Test preparation evokes images of teachers’ drilling students on irrelevant test questions 
for hours on end — bypassing the instructional program in order to engage in the dreaded 
“teaching to the test.” In an era of testing and accountability and current expert practice in 
schools, the reduction of the testing issue to a simplistic debate between “testing as savior” and 
“testing as evil” does not hold. We observed a variety of test preparation activities that 
supplemented rather than supplanted the instructional programs at our schools. We organized our 
findings into four categories of test preparation activities: (a) curriculum-embedded activities, (b) 
test practice, (c) environmental design, and (d) community outreach. 

Curriculum-embedded activities. Curriculum-embedded approaches to test preparation 
integrate the conceptual content of standardized tests into the regular instructional program. This 
strategy hinges on the argument that what is tested is worth learning in schools — with the 
corollary that what is tested can, if necessary, determine the range of taught content. As one of 
our principals explained, embedding standardized content in curriculum is an outcome of a 
design strategy for aligning curriculum, standards, and evaluation: 

We honestly provide them with the curriculum that the board has approved, and you know, our 
logic is that if the test is aligned to the standards and our curriculum is aligned with the standards, 

. . . [then] the curriculum is aligned with the test. [And if] the curriculum is aligned with the test, 
... we should be fine. 

Another principal described how integrating the tested content into the curriculum helped 
students make a seamless transition from course content to tested content: 
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Just by teaching them appropriately every single day in class, that’s how they are prepared. We’re 
going to teach them the best way we can every day in school, and hopefully that will get them 
where they need to be with testing. 

In data disaggregation and item analysis meetings across schools, we saw how teachers 
and leaders related exam and lesson content. At Pearson School, for example, teachers were 
particularly interested in understanding how the problems that surfaced through testing could be 
translated into lesson design. In fact, the Pearson School data reflection discussions were marked 
by contrasting agendas: the principal emphasized analyzing the results of testing, whereas 
teachers consistently moved from the results to how they could address the emergent issues in 
their classrooms. These discussions, and others, demonstrated how test preparation could be seen 
as a microcosm for the DDIS process as a whole. 

Test practice. Test practice focuses on helping students become familiar with the 
structure and format of the test by practicing test-taking skills outside the regular classroom 
curriculum. Test practice ranges from providing actual practice on similar tests to making sure 
students have an understanding of what the test structure will be. We discovered that our schools 
focused on developing test-taking skills rather than engaging in test drill. One principal declared: 
“We don’t teach them the test. Research has shown it’s not an effective way to get students to do 
well, and so we are not going to waste our time with that.” The schools developed testing skills 
in several ways. One school provided planning time for classroom teachers while other teachers 
taught a specific testing skill to the classes; another created a program in which some .skills 
preparation was done by guidance counselors in a Middle School and Beyond program. 

Environmental design. Environmental design employs strategies for creating a positive 
environment for students during testing. This environment includes not only what is done on the 
day of testing, but also events leading up to the testing. The strategies we observed included 
grouping students, organizing the testing environment, proctoring, and creating an exciting and 
healthy atmosphere for the tests. One principal discussed how she and her staff use grouping 
strategies to prepare students for testing: 

When we’re testing, we’ll do small groups. Some will be whole class, but in other situations we 
will have . . . special ed kids or ESL students, a small group with a specialist or an educator. And 
that time during testing . . . — it’s usually like a 2-week period in November — we will cancel all 
of our specialists’ [responsibilities], and they will assist with testing. So it’s a school-wide effort; 
all the teachers lose their assistant time. So everybody in the community is involved and assists 
during testing time. 

Student grouping also provides room for the school to use special testing strategies allowed by 
state regulations, such as reading exams aloud or providing unlimited time for certain groups of 
students to finish the exams. 

Coordinating a proctoring schedule is another environmental design task for the 
leadership team. One principal noted how encouraging teachers from across the school to proctor 
helped staff understand the rhythm and the demands of testing on children, as well as the content 
of the test. Another principal discussed how she prepared the proctoring plan: 
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I’ll go through it with my assistant principal, he’ll go through it with her [i.e., the program 
implementor], hut she’s the gatekeeper of that, she’s the one who really manages the groupings 
and things like that. And the teachers help, obviously — she [the program implementor] asks for 
input, first of all, from the teachers. You know, who would see, who do you feel would need a 
small group, and then she gets that and compiles that information and then figures it out from 
there. 

Some schools try to create an exciting and healthy atmosphere for testing to help children 
understand the importance of the test. One school held a pep rally with the high school 
cheerleaders and band to kick off testing week, then provided healthy snacks and slippers to help 
students feel comfortable with the often grueling exam process. 

Community outreach. Community outreach practices refer to how a school informs its 
school community about testing requirements. Now that schools are held accountable for 
standardized testing results, leaders in our schools have created artifacts to facilitate 
communication with parents and the community. One official described district outreach efforts 
as follows: 

Well have newsletters that go home to parents . . . [that give] strategies to help. We say [what] 
we’ve done at school for test preparation. . . . We talk about the superficial “get your kids to bed 
at a certain time” so we let parents know . . . when the testing is, when the tests are coming up 
.... [One] principal there has a meeting at the beginning of the year with the parents — I think it’ s 
the first day of school — . . . [to] explain the district’s priorities [and] the building’s goals for the 
year, what testing things we need to work on, what she would like done to help with that. . . . 

[Our] belief is that testing and its importance must be communicated from day one to the parents 
and school community. 

Back-to-school and teacher conference nights also provide opportunities to prepare the 
community for testing. Teachers at one school explained how they have developed reports based 
on a combination of state exams and their formative tests to explain student performance so that 
parents will understand reports of school performance in the local papers. 

Discussion and Conclusion 

This preliminary run through the data we have collected over this past year has helped us 
clarify the DDIS conceptual framework and understand how well the framework describes data- 
driven decision-making practices in schools. The DDIS was able to capture a range of activities 
leaders and teachers use to translate achievement scores into classroom-level information. The 
transition we observed between DDIS functions helped leaders to reduce the data, to understand 
what it meant, and to design and measure interventions for improving teaching and learning. We 
saw how teachers played a key role across DDIS functions in pushing the data toward creating 
useful information in the classroom, and how formal leaders worked to keep school-level 
questions of student achievement at the forefront of discussions. Moving back and forth between 
classroom- and school-level uses of data seemed to both rely upon and create the conditions for 
vibrant professional communities in each school. 

There are two areas for further investigation we would like to mention: subject matter 
leadership and the influence of special education practices. 
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We found significant subject matter differences in the capacity to use data for instruction 
across schools. In language arts, each school had developed a sophisticated, formative 
assessment model to gauge student progress. The capacity to generate useful data on student 
learning for teachers also seems to have sparked strong professional communities around 
language arts instruction in each school. In math, however, it was a different story. We have 
collected evidence that our schools are considering achievement data in math, but we have not 
seen the same commitment to local program design and formative assessment that we found in 
language arts. Each of our schools followed a district initiative to adopt a common math 
curriculum (Direct Instruction math in one school. Connected Math in another, and Everyday 
Math in the remaining two), but we observed little evidence of data use or professional 
community around the complexities of math teaching or learning. The math curriculum is, 
however, discussed in several of the schools as a form of outreach through workshops designed 
to teach parents how to experiment with math concepts with their children. Science and social 
studies instruction received less attention than either language arts or math. Indeed, even when 
schools discussed other topics in data disaggregation and item analysis sessions, teachers tended 
to consider science and social studies topics only in relation to reading comprehension and 
vocabulary familiarity. One school addressed science education as a part of a larger district 
initiative; another used a general curriculum design process to guide grade-level development of 
social studies and science projects; in yet another school, teachers talked about science in terms 
of the comprehension sections of the state exam. Understanding how subject matter influences 
the capacity to use data for instruction will be a central interest for our next round of research. 

Special education practices seemed to act as a template for organizing student-level 
program interventions. Each school had developed procedures to continuously generate data on 
program effectiveness. However, the action plans they developed upon realizing where students 
were falling short seemed to be organized for individual students on a special education student 
identification model. The familiar lEP, student-based intervention model helps school leaders 
and teachers focus their efforts on how to help individuals, but it may not address the systemic 
causes of why students are learning as they are. In our schools, faculty discussions about 
remediation — even in programs with strong evidence of data use — tended toward the approach 
of “how can we help this individual student?” rather than “how is our program leading toward 
this type of learning?” While the special education precedent may make discussions of student 
achievement more manageable, focusing accountability discussions on student-level rather than 
program-level issues may cause leaders to overlook the need for systemic overhaul. Critics of 
testing have commented on how student-level responses to accountability can lead to “gaming” 
the system by focusing on the needs of students on the cusp of proficiency rather than the needs 
of all students (Booher-Jennings, 2005; Jones, Jones, & Hargrove, 2003). We observed an 
interesting balance of curriculum-level and student-level interventions across our schools. 
Schools seemed to settle on program-level interventions, then use student- level programs to fine- 
tune program adequacy. Without feedback loops to integrate individual student interventions 
back into the program, however, the two levels of design could operate independently rather than 
as complementary pieces of an instructional system. Exploring the pervasiveness of the special 
education model may tell us much more about the constraints and affordances of processes 
schools develop to use data for instruction. 

We found it difficult to develop a clear, one-to-one relationship between functions, 
artifacts, and practices in the schools. The functions of the DDIS rarely occur in isolation, and 
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are linked together in complex organizational practices. One hypothesis could be that, from a 
design perspective, these functions originally correspond to specific artifacts, but as the DDIS 
becomes a mature system, the functions begin to merge across reflection, alignment, and design 
activities. We felt it was important to disentangle these functions for the purpose of guiding new 
leaders in building the pieces of a DDIS in their schools, but in future research, we will need to 
do more to determine the degree to which these functions are or should be linked together in 
received or created artifacts. Faced with the challenge of making improvements “all at once,” 
practitioners need to understand how functions can be assembled into systems that generate their 
own self-correcting feedback. 

Teaching to the test is a phrase that has long been used to criticize the role of testing in 
schools, implying that the content of schooling will be reduced to formulaic instruction. It is 
clear from our data that leaders have constructed multiple artifacts to link their instructional 
programs to the content and structure of tests. However, the DDIS we observed in action did not 
appear to have the effect of narrowing the curriculum to the structure and content of the test; 
rather, we observed rich instructional systems in the process of being redesigned to help children 
meet state standards as measured by state exams. The phrase teaching for the test better describes 
the instructional outcome achieved with a DDIS in place. A well-assembled DDIS links the 
different functions of the system to facilitate an information flow that serves improved student 
learning. Understanding this process offers insight into the central tasks and innovative practices 
of the new instructional leadership. 
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Appendix A 

DDIS Interview Protocol 

1. Overview of school programs: 

□ Which programs has the district/school invested in to improve the quality of 

teaching? 

□ Which programs has the district/school invested in to improve the quality of 

learning? 

□ What kinds of teacher committees and leadership committees exist in the school? 

□ What kinds of student support services are provided in your school? (e.g., Special 
Education, ESL, Talented and Gifted programs, etc.) 

i. Are these programs integrated into the regular school program, or are 
services provided outside the regular classroom? 

2. Data acquisition: 

□ What kinds of data do you rely on for improving learning? 

□ Describe the ways in which your school acquires information about student 
learning. 

□ Who works with those systems? Who helped build the systems, and who helps 
maintain them? 

3. Data reflection 

□ What practices or events guide the school community to make sense of 
achievement data? 

i. Are there established occasions to discuss disaggregated data? Are 
disaggregated data presented at the student level? At the classroom level? 
At the school level? 

ii. Are there occasions for teachers and staff to engage in item analysis of 
standardized tests? 

□ Which faculty and staff discuss data? How are these discussions structured? 

4. Program alignment: 

□ What efforts have been made to align instructional programs with learning and 
assessment goals? 

□ What materials and resources do you provide or use to guide curriculum 
alignment efforts? 

□ Who builds and maintains these materials and resources? 

5. Program design: 

□ Is each teacher held responsible for improving instruction to meet learning and 
assessment goals? 

□ Are there structures in place for teachers (at grade level or subject level) to 
collaboratively design instruction to meet learning goals? 

□ What are the main instructional design initiatives in math? In science? 

□ Who initiates and monitors instructional design efforts? 
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6. Formative feedback: 

□ How do teachers and leaders know whether their instructional design efforts are 
effective? 

□ Are there programs or structures in place that use feedback to reshape teaching 
and learning? 

i. In math? 

ii. In science? 

□ Who develops and maintains feedback practices? 

7. Test preparation: 

□ How are children prepared for testing? 

□ What processes and resources structure preparation practices? 

□ Who builds and maintains these processes and structures? 

8. Conclusion: 

□ Are there other insights/ideas/observations that we missed in our interview that 
you would like to share about data use in your school? 
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Appendix B 
DDIS Coding Scheme 

Revised 7/7/05 

Data acquisition — '"Processes used by schools to seek out, collect, and prepare information to 
guide teaching and learning. ” 

• Process 

■ Collection 

■ Storage — from filing cabinets to data warehouses 

■ Reporting — formal or informal reports, responses to queries 

• Agent (code in addition to process) 

■ Internal — done by the school 

■ External — done by an outside entity, such as district, state. Northwest Evaluation 
Association 

Data reflection — “Structured processes to engage the school community in making sense of 
student learning data that result in goals for improving teaching and learning. ” 

Question: What’s going on? Eocus on the students. 

• District led 

• School led 

• Grade-level led 

• Content area led 

Program alignment — “Program alignment processes make the school ’s curriculum congruent 
with relevant content and performance standards as well as what is taught in classrooms in 
order to improve student learning. ” 

Question: What do we need to change? (generally will use multiple codes) 

• Standards — set by district and state 

• Practice — what is taking place in the classroom 

• Curriculum — what is to be taught 

• Assessment — how student progress is measured 

Program design — “How a school acts on perceived instructional needs through the creation or 
adaptation of curricula, pedagogies, student service programs, and instructional strategies in 
order to improve student learning. ” 

Question: How are we going to change? (Always code for source in addition) 

• Curriculum-focused design 

• Student-focused design 

• Source 

■ Eocal design — modified or created locally 

■ Received — bought or given from outside 

■ Inherited — predates the principal 
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Formative feedback — “Learner-focused iterative evaluation cycles designed to create ongoing 
timely flows of information used to improve both student learning and instructional program 
quality across the school. ” 

Question: Are our changes working as we want them to? 

• Program level 

• Administrator level 

• Teacher level 

• Student level 

• Positive feedback 

Test preparation — “Activities designed to motivate students and to develop strategies for 
improving performance in taking state and district assessments. ” 

• Community outreach 

• Embedded in curriculum 

• Environmental design 

• Test practice 
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